System and method for multisensory psychomotor skill training

ABSTRACT

A system and method for delivering psychomotor skill training, providing didactic instruction using an internet portal, tracking performance, and providing user feedback within a multisensory environment, serving as an integration for existing multimodal ultrasound training systems. This multisensory psychomotor skill training environment of the present invention would extend the existing described patented technologies by further adding realism through representation of virtual (digital) characters into a training scenario (e.g., pregnant patient), superimposing virtual (digital) guides or prompts onto actual real physical objects (e.g., anatomical landmarks or guides onto a mannequin or patient) embedding the training instruments within a broader virtual (digital) scenario (e.g., operating room). The ability to create these virtual, augmented, and mixed reality scenarios enables cost-effective, realistic, and improved training capabilities that mirror real-life training scenarios.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/880,290, filed Jan. 25, 2018, titled “System and Method forMultisensory Psychomotor Skill Training,” which claims the benefit ofU.S. Provisional Patent Application No. 62/450,975, filed Jan. 26, 2017,which applications are incorporated here in their entirety by thisreference.

TECHNICAL FIELD

This invention relates generally to systems and methods for providingmedical training, and more specifically to medical training systems andmethods that at least partly involve simulations of medical proceduresand operations, particularly ultrasonography.

BACKGROUND

Paradigm shifts are taking place in the world of medicine andtechnology. Point-of-care ultrasonography is redefining the very essenceof the physical examination and the way clinicians practice medicine.Point-of-care ultrasound refers to the use of portable ultrasonographyat a patient's bedside for diagnostic (e.g., symptom or sign-basedexamination) and therapeutic (e.g., image-guidance) purposes. Theprincipal barrier to use of this life-saving technology is lack ofultrasound training. User adoption is limited by the psychomotor skillsrequired to manipulate an ultrasound probe coupled with the ability tointerpret the resulting ultrasound images. Concurrent with improvingpsychomotor skills, training must also reflect the psychosocialstressors associated with performing procedures. It is clear thatmedical education must catch up with the expanded use of ultrasonographyif tomorrow's practitioners are to provide superior healthcare for theirpatients.

Developing competence on performing ultrasonography in a clinicalsetting requires integrated cognitive (image interpretation) andpsychomotor (optimal image window acquisition) skills and developing theability to perform these procedures under stress. Once an optimal imagewindow is acquired and correctly interpreted, the information needs tobe correctly applied to patient care. The opportunity cost of traininghealthcare providers on ultrasonography is extremely high. Optimaltraining requires: (1) a qualified instructor; (2) trainees; (3) anultrasound machine; (4) a patient with a pathologic condition; and (5)an environment that accurately recreates the psychosocial challenges ofperforming procedures. All of these elements must come together in thesame place and at the same time.

Currently available training methods all have significant limitations.These include clinical bedside teaching, attending hands-on trainingcourses, use of phantom models, and high-fidelity yet very expensiveultrasound simulator workstations. They may involve bulky trainingplatforms that require multiple users to visit a simulation center(e.g., UltraSim®, CAE Healthcare) or require the presence of an actualultrasound machine (e.g., Blue Phantom™). These ultrasound-trainingsolutions employ high-priced dedicated computer hardware and softwarethat does not deploy over the Internet. Alternative training productsprovide a limited library of purely didactic training solutions that arenot accompanied with any hands-on training experience (e.g., EMSONO).None of the above solutions recreate the psychosocial stressors ofperforming procedures in a real-life clinical setting.

Conversely, within the fields of virtual reality, augmented reality, andmixed reality technologies there exists the ability to create virtual(digital) characters, environments, and guides that can be superimposedon actual physical objects. However, none of these technologies offerthe ability to develop and train individuals on highly refinedpsychomotor skills, such as performing an ultrasound-guided procedure.

For the foregoing reasons there is a need for a system and method todevelop psychomotor skills of individuals on a wide variety of tasks,such as ultrasound-guided procedures, in a context (or clinical setting)that accurately imposes the psychosocial stressors of performing thesestated procedures in a real clinical setting. Individuals skilled inthese arts and technologies have not been able to devise a solution tothe described limitations of existing training solutions. The lack ofexisting products and/or technology solutions to this problem, at thetime of this filing, speaks to the lack of a clear and/or obvioussolution to this problem.

SUMMARY

The present invention is directed to creating the perception of presencein a multisensory psychomotor skill training environment that wouldextend the existing described patented technologies by further addingrealism through representation of virtual (digital) characters into atraining scenario (e.g., pregnant patient), superimposing virtual(digital) guides or prompts onto actual real physical objects (e.g.,anatomical landmarks or guides onto a mannequin or patient), orembedding the training instruments within a broader virtual (digital)scenario (e.g., operating room).

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows a schematic flow diagram illustrating how the variouscomponents of the system of the present invention may interact.

DETAILED DESCRIPTION OF THE INVENTION

The detailed description set forth below in connection with the appendeddrawings is intended as a description of presently-preferred embodimentsof the invention and is not intended to represent the only forms inwhich the present invention may be constructed or utilized. Thedescription sets forth the functions and the sequence of steps forconstructing and operating the invention in connection with theillustrated embodiments. It is to be understood, however, that the sameor equivalent functions and sequences may be accomplished by differentembodiments that are also intended to be encompassed within the spiritand scope of the invention.

The present invention is a system and method for delivering psychomotorskill training to provide didactic instruction for performing a skill inat least a partially virtual environment. In some embodiments, using aninternet portal, tracking performance, and providing user feedbackwithin a multisensory environment integrates existing multimodalultrasound training systems with the psychomotor skill training system.The feedback may be in the form of an auditory feedback, visualfeedback, or tactile feedback to alert or instruct the user regardinghis or her performance.

The existing multimodal ultrasound training systems comprise a widevariety of immersive and non-immersive visualization technologies thathave been described as virtual reality, augmented reality, and mixedreality. U.S. Pat. No. 8,480,404 to Savitsky (the '404 patent) and U.S.Pat. No. 8,297,983 to Savitsky et al. (the '983 patent) describemultimodal ultrasound training systems in greater detail. The '404 and'983 patents are incorporated in their entirety here by this reference.

The '404 patent teaches a medical procedure training system based on acomputer platform that provides multimodal education within a virtualenvironment. The system integrates digital video, three-dimensionalmodeling, and force-feedback devices for the purpose of training medicalprofessionals on the skills required to perform medical procedures. Itcan be used to teach a variety of medical procedures and interventions,including ultrasound. The '983 patent describes an ultrasound trainingsystem having a data capture module, a digital asset management module,a validation module, didactic content, a media asset production module,an integration layer, an internet-based portal, a software client, and aperipheral probe. The combination of the disclosed elements creates asystem that enables the creation and delivery of high-quality ultrasoundeducation and training in a low-cost, widely deployable, and scalablemanner, with a facilitated method for processing orders and financialtransactions between customers and content providers.

The multisensory psychomotor skill training environment of the presentinvention would extend the existing described patented technologies byfurther adding realism through representation of virtual (digital)characters into a training scenario (e.g., pregnant patient),superimposing virtual (digital) guides or prompts onto actual realphysical objects (e.g., anatomical landmarks or guides onto a mannequinor patient), or embedding the training instruments within a broadervirtual (digital) scenario (e.g., operating room). The ability to createthese virtual, augmented, and mixed reality scenarios enablescost-effective, realistic, and improved training capabilities thatmirror real-life training scenarios.

Importantly, the present invention would eliminate the need for physicalstructures (e.g., hospital beds, manikins, etc.) within the trainingscenario, as these elements can be virtually represented—making trainingscalable, cost-effective, and efficient. Combining the virtual,augmented, and mixed reality scenarios with the multimodal ultrasoundtraining system enables effective ultrasound training for performance offine psychomotor skill-dependent skills with the added psychosocialstressors associated with performing these procedures in real clinicalsettings. This proposed system and method for providing multisensorypsychomotor skill training will address the unmet need for a method todevelop the psychomotor skills of individuals on a wide variety oftasks, such as ultrasound-guided procedures, in in a realistic clinicalscenario with proportionate psychosocial stressors.

The present invention hinges on emerging technologies capable ofcreating the perception of presence to enhance the psychological impactand the effectiveness of medical training simulators. The notion ofpresence for our purpose has two fundamental meanings:

(1) the cognitive illusion of being truly transported into an alternateworld recreated by computer simulations; or

(2) the cognitive illusion that an object, a person, or otherinteractive medium generated by computer simulations is truly present inthe surrounding world perceived by the user.

The notion of presence is fundamentally important in its ability toinstill a powerful emotional response in users—an essential requisitefor our invention. Collectively, these technologies are defined asprojector of presence (“POP”) devices. The POP devices address theproblem associated with recreating the visual and auditory stimuli thatplay a primary role in creating the perception of presence andsynchronizing those stimuli with the reality that surrounds the user. Apractitioner skilled in the art will recognize that several commerciallyavailable POP devices exist already. POP devices are grouped in severalbroad categories:

(1) Virtual Reality (VR), wherein the sight of the real world isentirely blocked by an opaque medium and the user's field of view iscompletely replaced by a rendering of a simulated reality (virtualworld) in which solutions generally rely on sophisticated motion sensingtechnologies to synchronize the rendering of the virtual world with theuser's head motion and in some cases his or her position in the realworld;

(2) Augmented Reality (AR), wherein the content is overlaid on theuser's field of view of the real world and enhanced with audio thatresponds to the surrounding real world in which true AR solutions aregenerally restricted to showing notifications or graphical userinterfaces superimposed on real objects; or

(3) Mixed Reality (MR), wherein solutions extend the basic notion of ARby synchronizing the content overlaid onto the user's field of view withthe surrounding real world, thus recreating a convincing perception thatthe simulated objects are really present in front of the user.

While the present invention can be realized with any of these categoriesof POP devices, the preferred embodiment utilizes VR and MR for theirenhanced ability in creating the perception of presence. Further, it hasbeen shown that for many applications, tactile feedback is also veryimportant in establishing a strong perception of presence, especially insituations where a simulated real world task does in fact requireinteraction with a physical object. The present invention conveys thestrongest perception of presence and succeeds in recreatingpsychological stressors that occur in real-world clinical scenarios.

Unless otherwise stated, the following terms as used herein have thefollowing definitions.

“Projector of Presence” of “POP” devices refers to any VR, AR, or MRdevice capable of measuring key aspect of the user's head position, bodyplacement, and gaze direction using a first set of sensors to measureposition, orientation, and/or motion in order to project realisticrenditions of individual objects or entire settings into the user'sfield of view. These types of technologies exist already and are rapidlyevolving.

“Tracked probe” refers to a device preferably in the shape of a medicalinstrument with a second set of sensors to measure its position,orientation and/or motion, and with communication hardware to transmitsensor information to the computer.

“Scanning surface” refers to a planar or curvilinear solid surface onwhich the user can place and move the tracked probe on top of, such as amedical manikin. Preferably the geometry of the scanning surface isknown by the simulation software so that the tactile feedback of thetracked probe moving on the scanning surface can be reproducedaccurately in the virtual environment projected by the POP device. Thescanning surface may contain a third set of sensors so that itsposition, orientation, and/or motion can be measured and detected.

The sensors for detecting position, orientation, and/or motion can berealized using MEMS sensors, such as accelerometers, gyroscopes, andmagnetometers for self-contained solutions, or the can be built usingexternal optical or electromagnetic trackers that act as position andorientation references. Other sensing solutions capable of measuringposition, orientation and/or motion fall within the spirit of thisinvention.

A “computer” refers to any type of desktop, laptop, tablet, mobilecomputer, or any other computational device capable of running thesimulation software and/or communicating through a wired or wirelessconnection with the POP device and tracked probe.

A “simulation software,” runs on the computer and provides a realisticrepresentation of a clinical scenario to train students on the use ofmedical imaging and how to perform certain types of procedures. Severaltypes of simulation solutions exist on the market and demonstrate theimportance of this type of technology. However, current solutions arenot capable of instilling the same types of psychosocial stressors thata medical practitioner encounters in real life or emulating interactionswith other medical professionals involved in a clinical scenario (e.g.nurses and radiologists).

The simulation software augmented with a POP device renders and animatesa clinical scene and provides a convincing rendition of a patient inpain that emotes vigorously during a procedure. The simulation softwarealso renders and animates other medical professionals interacting andtalking to the user to help him or her understand the team effortrequired to diagnose and treat a patient in real life and to emulate thepsychological pressure of being surrounded by other individuals thatmust coordinates their efforts to achieve a desired clinical goal. Thecomputer monitors the position, orientation, and movement of the trackedprobe to determine whether the user has performed the proper procedurewith a predetermined level of proficiency by determining whether thetracked probe was in the correct position and orientation and moved in apredetermined manner within a specified tolerance level for a givenscenario.

As illustrated in FIG. 1 , the present invention, in the preferredembodiment, comprises a POP device, a tracked probe emulating a medicalimplement whose orientation and/or position can be tracked in 3D space,an optional scanning surface that emulates the physicality of a humanbody or a section of it, a computer capable of running the simulation,and software running on the computer that gathers data from the variousdevices comprising this invention and actuates them based on the stateof the simulation.

In use, a user wears and activates a POP device. A computer incommunication with the POP device generates a simulation in the user'sfield of view. The simulation may be all the user sees in the user'sfield of view or the simulation may be integrated with the real worldenvironment in the user's field of view. The user holds a tracked probein the user's hand, and when brought into the user's field of view, thetracked probe enters the simulation. As the tracked probe interacts withthe simulation, the tracked probe provides tactile feedback thatcontributes to a perception of the user.

In some embodiments, the tracked probe may be used with a scanningsurface. The tracked probe interacts with the scanning surface and thecombined interaction provides a tactile feedback that contributes to theperception of the user.

In some embodiments, the computer in communication with the POP deviceand the tracked probe determines the position and orientation of thetracked probe and the POP device and projects a simulated environment onthe POP device based on the position and orientation of the POP deviceand the tracked probe. The POP device uses the sensors to measureinformation, such as the orientation and/or position of the user's headand gaze direction, and reports the information to the computer.

The computer may be integrated with the POP device or may be a separatesystem that communicates with the POP device and supports the processingrequirements of the POP device. The computational system uses theinformation to generate a rendering of a simulated scene in the POPdevice for display in the user's field of view. The simulated scene mayrepresent a clinical scenario or individual objects in a way thatmatches the user's viewpoint and the current state of the tracked probe.

The POP device renders an image, and optionally auditory feedback, toconvince the user that the simulated scene generated by the computersimulation is truly present in the user's field of view. By way ofexample only, in an ultrasound training application, the POP device maypresent a simulated hospital scene in which the tracked probe isrepresented as an ultrasound probe. The scanning surface may bepresented as a patient. As the user looks around the room, the POPdevice generates a simulated monitor when the user looks in a specificlocation. The specific location may be predetermined, or the user mayselect a particular location to generate the simulated monitor. The usercan then begin his or her ultrasound examination on the scanningsurface. If the user's gaze is directed towards the tracked probe andscanning surface, then the user will see the ultrasound probe andpatient. If the user turn's his or her head in the particular locationwhere the monitor was established, then the user sees on the simulatedmonitor, an ultrasound image. As the user manipulates the ultrasoundimage along the scanning surface, the ultrasound image changesaccordingly giving the user the perception of actually conducting anultrasound scan.

The simulation software can create a scene in which the user mustperform the proper procedure to pass the test. Whether the user passesthe test will be determined by the position, orientation, and/ormovement of the tracked probe. Timing can also be integrated into thescene by allowing the user to perform a particular procedure at aspecified time that the user must recognize. Distractions can also beincorporated into the scene to determine whether the user is able tomaintain focus.

A database of scenes may be stored on the computer or other storagedevice for the user to select to test various skills and knowledge.

The system can take the form of an entirely hardware embodiment, anentirely software embodiment or an embodiment containing both hardwareand software elements. In one embodiment, the system is implemented insoftware, which includes but is not limited to firmware, residentsoftware, microcode, etc.

Furthermore, the system can take the form of a computer program productaccessible from a computer-usable or computer-readable medium providingprogram code for use by or in connection with a computer or anyinstruction execution system. For the purposes of this description, acomputer-usable or computer readable medium can be any apparatus thatcan contain, store, communicate, propagate, or transport the program foruse by or in connection with the instruction execution system,apparatus, or device.

The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) or apropagation medium. Examples of a computer-readable medium comprise asemiconductor or solid-state memory, magnetic tape, a removable computerdiskette, a random access memory (RAM), a read-only memory (ROM), arigid magnetic disk and an optical disk. Current examples of opticaldisks comprise compact disk-read only memory (CD-ROM), compactdisk-read/write (CD-R/W) and DVD.

A data processing system suitable for storing and/or executing programcode comprises at least one processor coupled directly or indirectly tomemory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories that provide temporary storage of at leastsome program code in order to reduce the number of times code isretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems orremote printers or storage devices through intervening private or publicnetworks. Modems, cable modem and Ethernet cards are just a few of thecurrently available types of network adapters.

Described above, aspects of the present application are embodied in aWorld Wide Web (“WWW”) or (“Web”) site accessible via the Internet. Asis well known to those skilled in the art, the term “Internet” refers tothe collection of networks and routers that use the Transmission ControlProtocol/Internet Protocol (“TCP/IP”) to communicate with one another.The internet can include a plurality of local area networks (“LANs”) anda wide area network (“WAN”) that are interconnected by routers. Therouters are special purpose computers used to interface one LAN or WANto another. Communication links within the LANs may be wireless, twistedwire pair, coaxial cable, or optical fiber, while communication linksbetween networks may utilize analog telephone lines, digital T-1 lines,T-3 lines or other communications links known to those skilled in theart.

Furthermore, computers and other related electronic devices can beremotely connected to either the LANs or the WAN via a digitalcommunications device, modem and temporary telephone, or a wirelesslink. It will be appreciated that the internet comprises a vast numberof such interconnected networks, computers, and routers.

The foregoing description of the preferred embodiment of the inventionhas been presented for the purposes of illustration and description. Itis not intended to be exhaustive or to limit the invention to theprecise form disclosed. Many modifications and variations are possiblein light of the above teaching. It is intended that the scope of theinvention not be limited by this detailed description, but by the claimsand the equivalents to the claims appended hereto.

What is claimed is:
 1. A system for medical training, comprising: a) aPOP device configured to register a position and orientation of the POPdevice with a computer using a first set of sensors; b) a probeconfigured to register a position and orientation of the probe with thecomputer using a second set of sensors; c) a scanning surface configuredto register a position and orientation of the scanning surface with thecomputer using a third set of sensors; and d) the computer configured tomonitor the position and orientation of the POP device, the position andorientation of the probe, and the position and orientation of thescanning surface; e) the computer programmed with a simulation softwarethat when executed: i) generates a simulated environment in the POPdevice representing a clinical scenario in a manner that correspondswith a user's gaze direction; ii) generates a virtual medical instrumentbased on the position and orientation of the probe; and iii) generates avirtual patient based on the position and orientation of the scanningsurface.
 2. The system of claim 1, wherein the computer is configured toprovide feedback to the user.
 3. The system of claim 2, wherein thefeedback is audio feedback through the POP device.
 4. The system ofclaim 2, wherein the feedback is visual feedback through the POP device.5. The system of claim 1, wherein the probe is configured to generatetactile feedback corresponding with a scene in the simulatedenvironment.
 6. The system of claim 1, wherein the simulation softwareis configured to animate a clinical scene and the computer is configuredto determine a level of proficiency of the user by determining whetherthe probe was in a correct position and orientation, and moved in apredetermined manner within a specified tolerance level.
 7. The systemof claim 1, wherein the computer is configured to track a performance ofthe user.
 8. The system of claim 1, wherein the POP device is selectedfrom the group consisting of a virtual reality device, an augmentedreality device, and a mixed reality device.
 9. A method for medicaltraining, comprising the steps of: a) a POP device registering aposition and orientation of the POP device with a computer; b) a proberegistering a position and orientation of the probe with the computer;c) a scanning surface registering a position and orientation of thescanning surface with the computer; d) the computer monitoring theposition and orientation of the POP device, the position and orientationof the probe, and the position and orientation of the scanning surface;e) the computer programmed with simulation software that when executed:i) generates a simulated environment in the POP device representing aclinical scenario in a manner that corresponds with the position andorientation of the POP device; ii) generates a virtual medicalinstrument based on the position and orientation of the probe; and iii)generates a virtual patient based on the position and orientation of thescanning surface.
 10. The method of claim 9, wherein the computerprovides feedback to the user.
 11. The method of claim 10, wherein thefeedback is audio feedback through the POP device.
 12. The method ofclaim 10, wherein the feedback is visual feedback through the POPdevice.
 13. The method of claim 10, wherein the probe generates tactilefeedback corresponding with a scene in the simulated environment. 14.The method of claim 13, wherein the simulation software animates aclinical scene and the computer determines a level of proficiency of theuser by determining whether the probe was in a correct position andorientation, and moved in a predetermined manner within a specifiedtolerance level.
 15. The method of claim 14, wherein the computer tracksa performance of the user.
 16. The method of claim 15, wherein the POPdevice is selected from the group consisting of a virtual realitydevice, an augmented reality device, and a mixed reality device.
 17. Amethod for medical training, comprising the steps of: a) displaying atleast a partially simulated environment in a POP device based on aposition and orientation of the POP device; b) displaying a virtualmedical instrument in the at least partially simulated environment basedon a position and orientation of a probe when the POP device is directedtowards the probe; c) displaying a virtual patient in the at leastpartially simulated environment based on a position and orientation of ascanning surface when the POP device is directed towards the scanningsurface; and d) displaying a simulated monitor in the at least partiallysimulated environment when the POP device is directed towards apredetermined location.
 18. The method of claim 17, further comprisinggenerating tactile feedback to the probe while the probe is moved alongthe scanning surface.
 19. The method of claim 18, displaying anultrasound image in the simulated monitor corresponding to the positionand orientation of the probe relative to the position and orientation ofthe scanning surface when the POP device is directed towards thepredetermined location.